Zillow MMM - Submits
Spencer Smith
Generated on ‘13 February, 2020’
Model Hyper Parameters
Training dates 2016-09-01 to 2019-12-30
Transformations
| Channel | Diminishing Returns Shape | Decay rate |
|---|---|---|
| TV | log | 0.98 |
| Digital | 0.9 | 0.3 |
| SEM Brand | 0.2 | 0.4 |
| SEM Non-Brand | 0.9 | 0.4 |
| Social | 0.9 | 0.6 |
| App | 0.9 | 0.93 |
Model Performance
\(R^2\) = 0.891834
MAPE = 4.17%
MAPE (Test) = 4.34%
RMSE = 2,377
RMSE (Test) = 2,426
| Estimate | t value | Pr(>|t|) | |
|---|---|---|---|
| (Intercept) | -962.624 * | -0.377 | 0.706 |
| tv | 6.539 | 2.946 | 0.003 |
| semBR | 154.551 | 2.409 | 0.016 |
| semNB | 0.039 | 2.648 | 0.008 |
| app | 0.022 | 4.560 | 0.000 |
| social | 0.063 | 4.449 | 0.000 |
| NA | NA | NA | NA |
| email_marketing | 0 * | -0.727 | 0.467 |
| email_operational | 0.001 | 5.338 | 0.000 |
| email_savedsearch | 0 * | 0.219 | 0.827 |
| SEO_bigcity | 2659.025 | 3.037 | 0.002 |
| SEO_hdp | 2409.648 * | 1.719 | 0.086 |
| push_android | 0 | -2.775 | 0.006 |
| push_ios | 0 | 3.089 | 0.002 |
| NA.1 | NA | NA | NA |
| nonbrand_queries | 0.308 | 16.008 | 0.000 |
*Not significant effect
Lift Analysis
Lift is the expected contribution of the channel to the overall prediction. “If we removed the channel, what percent of total Submits would we lose?”| 2017 | 2018 | 2019 | |
|---|---|---|---|
| (Intercept) | -2.1% | -2.2% | -2.2% |
| tv | 5.8% | 7.0% | 7.0% |
| semBR | 3.4% | 3.6% | 3.6% |
| semNB | 2.0% | 2.4% | 2.3% |
| app | 3.6% | 5.0% | 7.6% |
| social | 3.0% | 3.7% | 6.3% |
| digital | NA% | NA% | NA% |
| email_marketing | -0.2% | -0.1% | -0.1% |
| email_operational | 5.9% | 3.7% | 5.4% |
| email_savedsearch | 0.1% | 0.2% | 0.2% |
| SEO_bigcity | 4.3% | 5.1% | 3.4% |
| SEO_hdp | 3.6% | 3.5% | 3.0% |
| push_android | -1.9% | -1.3% | -1.9% |
| push_ios | 1.1% | 1.9% | 3.4% |
| active_listings | NA% | NA% | NA% |
| nonbrand_queries | 74.0% | 75.2% | 78.6% |
CPA Analysis
| date | tv | semBR | semNB | app | social | digital |
|---|---|---|---|---|---|---|
| 2016-09-01 | $36.96 | $5.06 | $45.57 | $8.01 | $16.42 | Inf |
| 2017-01-01 | $41.37 | $5.41 | $44.68 | $8.38 | $17.36 | Inf |
| 2018-01-01 | $38.80 | $5.84 | $45.32 | $8.83 | $17.71 | Inf |
| 2019-01-01 | $32.96 | $6.46 | $45.21 | $8.90 | $18.79 | Inf |
Error Plot
Variable Analysis
Variable Correlation
Media Marginal Plots
The “from June” color scale is meant to be used as a visual heuristic representing peak season. The equation is absolute value of the month number minus 6. \[from\_june = Abs(month - 6)\]
Variable plots
Owned
## [1] "email_marketing" "email_operational" "email_savedsearch"
## [4] "push_android" "push_ios"
Search
## [1] "SEO_bigcity" "SEO_hdp" "nonbrand_queries"
Apendix
Seasonal Variable
| Estimate | t value | Pr(>|t|) | |
|---|---|---|---|
| christmas | -9,454 | -9.768 | 0.000 |
| easter | -4,634 | -4.474 | 0.000 |
| fathers | -2,619 | -2.517 | 0.012 |
| halloween | -1,943 * | -1.540 | 0.124 |
| independence | -9,288 | -6.286 | 0.000 |
| labor | -2,367 | -3.145 | 0.002 |
| martin | 1,024 * | 0.712 | 0.477 |
| memorial | -3,532 | -3.998 | 0.000 |
| mothers | -3,206 | -3.069 | 0.002 |
| new | -6,421 | -6.185 | 0.000 |
| presidents | 3,076 | 2.134 | 0.033 |
| super | -665 * | -0.459 | 0.646 |
| thanksgiving | -4,192 | -5.882 | 0.000 |
| valentines | -4,883 | -3.408 | 0.001 |
| year2017 | -517 * | -1.387 | 0.166 |
| year2018 | -2,668 | -5.217 | 0.000 |
| year2019 | -6,796 | -8.855 | 0.000 |
| month2 | 2,355 | 5.135 | 0.000 |
| month3 | 1,665 | 3.446 | 0.001 |
| month4 | 1,423 | 2.465 | 0.014 |
| month5 | -1,938 | -3.012 | 0.003 |
| month6 | -1,776 | -2.720 | 0.007 |
| month7 | -608 * | -0.916 | 0.360 |
| month8 | -843 * | -1.612 | 0.107 |
| month9 | -2,669 | -5.789 | 0.000 |
| month10 | -3,595 | -8.345 | 0.000 |
| month11 | -4,956 | -11.094 | 0.000 |
| month12 | -7,087 | -15.731 | 0.000 |
| wdayMonday | 3,383 | 8.274 | 0.000 |
| wdaySaturday | -162 * | -0.454 | 0.650 |
| wdaySunday | -1,988 | -4.364 | 0.000 |
| wdayThursday | 1,680 | 4.756 | 0.000 |
| wdayTuesday | 2,723 | 7.104 | 0.000 |
| wdayWednesday | 2,542 | 6.954 | 0.000 |